Systems and methods for automated detection of an indication of malignancy in a mammographic image

ABSTRACT

There is provided a method of computing a likelihood of malignancy in a mammographic image, comprising: receiving a single channel 2D mammographic image including a single pixel intensity value for each pixel thereof, converting the single channel 2D mammographic image into a multi channel 2D mammographic image including multiple pixel intensity value channels for each pixel thereof, computing by a first sub-classifier according to the whole multi channel image, a first score indicative of likelihood of malignancy within the whole multi channel image, computing by a second sub-classifier according to each respective patch extracted from the multi channel image, a respective second score indicative of likelihood of malignancy within each respective patch, and computing by a gating sub-classifier according to the first score and the second scores, an indication of likelihood of malignancy and a location of the malignancy.

RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/701,543 filed on Sep. 12, 2017, which claims the benefit of priorityunder 35 USC § 119(e) of U.S. Provisional Patent Application No.62/393,180 filed on Sep. 12, 2016. The contents of the aboveapplications are all incorporated by reference as if fully set forthherein in their entirety.

FIELD AND BACKGROUND OF THE INVENTION

The present invention, in some embodiments thereof, relates tomammographic image processing and, more specifically, but notexclusively, to systems and methods for automated detection ofmalignancy in a mammographic image.

Breast cancer is the most prevalent malignancy in the US and the thirdhighest cause of cancer-related mortality worldwide. In the US over230,000 new diagnoses and approximately 40,000 deaths occur annually,for example, as described with reference to American Cancer Society,“Cancer Facts & Figures 2015,” Cancer Facts Fig. 2015, pp. 1-9, 2015.Regular mammography screening has been attributed with doubling the rateof early cancer detection and has been credited with decreasing breastcancer mortality by up to 30% over the past three decades, for example,as described with reference to B. Lauby-Secretan, C. Scoccianti, D.Loomis, L. Benbrahim-Tallaa, V. Bouvard, F. Bianchini, and K. Straif,“Breast-cancer screening—viewpoint of the IARC Working Group.,” N. Engl.J. Med., vol. 372, no. 24, pp. 2353-8, 2015, and A. M. Kavanagh, G. G.Giles, H. Mitchell, and J. N. Cawson, “The sensitivity, specificity, andpositive predictive value of screening mammography and symptomaticstatus.,” J. Med. Screen., vol. 7, no. 2, pp. 105-10, 2000. Yetestimates of mammographic accuracy in the hands of experiencedradiologists remains suboptimal with sensitivity ranging from 62-87% andspecificity from 75-91% for example, as described with reference to A.M. Kavanagh, G. G. Giles, H. Mitchell, and J. N. Cawson, “Thesensitivity, specificity, and positive predictive value of screeningmammography and symptomatic status.,” J. Med. Screen., vol. 7, no. 2,pp. 105-10, 2000, C. D. Lehman, R. D. Wellman, D. S. M. Buist, K.Kerlikowske, A. N. A. Tosteson, D. L. Miglioretti, and Breast CancerSurveillance Consortium, “Diagnostic Accuracy of Digital ScreeningMammography With and Without Computer-Aided Detection.,” JAMA Intern.Med., vol. 175, no. 11, pp. 1828-37, November 2015, N. S. Winkler, S.Raza, M. Mackesy, and R. L. Birdwell, “Breast density: clinicalimplications and assessment methods.,” Radiographics, vol. 35, no. 2,pp. 316-24, 2015, T. M. Kolb, J. Lichy, and J. H. Newhouse, “Comparisonof the Performance of Screening Mammography, Physical Examination, andBreast US and Evaluation of Factors that Influence Them: An Analysis of27,825 Patient Evaluations,” Radiology, vol. 225, no. 1, pp. 165-175,October 2002, and K. Kerlikowske, R. A. Hubbard, D. L. Miglioretti, B.M. Geller, B. C. Yankaskas, C. D. Lehman, S. H. Taplin, and E. A.Sickles, “Comparative effectiveness of digital versus film-screenmammography in community practice in the United States: A cohort study,”Ann. Intern. Med., vol. 155, no. 8, pp. 493-502, 2011.

SUMMARY OF THE INVENTION

According to a first aspect, a method of computing an indication oflikelihood of malignancy in a two dimensional (2D) x-ray based singlechannel mammographic image by a trained statistical classifier,comprises: receiving a single channel 2D mammographic image of at leasta portion of a breast, wherein the single channel 2D mammographic imageincludes a single pixel intensity value for each pixel of a plurality ofpixels thereof, converting the single channel 2D mammographic image intoa multi channel 2D mammographic image including a plurality of pixelintensity value channels for each pixel of a plurality of pixelsthereof, computing by a first sub-classifier of the trained statisticalclassifier according to the whole multi channel 2D mammographic image, afirst score indicative of likelihood of malignancy within the wholemulti channel 2D mammographic image, computing by a secondsub-classifier of the trained statistical classifier according to eachrespective patch of a plurality of patches extracted from the multichannel 2D mammographic image, a respective second score of a pluralityof second scores indicative of likelihood of malignancy within eachrespective patch of the plurality of patches, computing by a gatingsub-classifier of the trained statistical classifier according to thefirst score and the plurality of second scores, an indication oflikelihood of malignancy and a location of the malignancy, and providingthe indication of likelihood of malignancy and the location of themalignancy.

According to a second aspect, a system for computing an indication oflikelihood of malignancy in a two dimensional (2D) x-ray based singlechannel mammographic image by a trained statistical classifier,comprises: a non-transitory memory having stored thereon a code forexecution by at least one hardware processor of a computing device, thecode comprising: code for receiving a single channel 2D mammographicimage of at least a portion of a breast, wherein the single channel 2Dmammographic image includes a single pixel intensity value for eachpixel of a plurality of pixels thereof, code for converting the singlechannel 2D mammographic image into a multi channel 2D mammographic imageincluding a plurality of pixel intensity value channels for each pixelof a plurality of pixels thereof, code for computing by a firstsub-classifier of the trained statistical classifier according to thewhole multi channel 2D mammographic image, a first score indicative oflikelihood of malignancy within the whole multi channel 2D mammographicimage, code for computing by a second sub-classifier of the trainedstatistical classifier according to each respective patch of a pluralityof patches extracted from the multi channel 2D mammographic image, arespective second score of a plurality of second scores indicative oflikelihood of malignancy within each respective patch of the pluralityof patches, code for computing by a gating sub-classifier of the trainedstatistical classifier according to the first score and the plurality ofsecond scores, an indication of likelihood of malignancy and a locationof the malignancy, and code for providing the indication of likelihoodof malignancy and the location of the malignancy.

According to a third aspect, a method of training a statisticalclassifier for computing an indication of likelihood of malignancy in a2D x-ray based single channel mammographic image, comprises: receiving aplurality of single channel 2D mammographic training images, each 2Dmammographic training image of the plurality of 2D mammographic trainingimages including of at least a portion of a breast, wherein each singlechannel 2D mammographic training image includes a single pixel intensityvalue for each pixel of a plurality of pixels thereof, wherein eachsingle channel 2D mammographic training image of the plurality of singlechannel 2D mammographic training images is associated with a positiveindication of malignancy or a negative indication of malignancy, whereineach member of a sub-set of the plurality of single channel 2Dmammographic training images associated with the positive indication ofmalignancy is further associated with a location of the malignancy,converting the plurality of single channel 2D mammographic trainingimages into corresponding plurality of multi channel 2D mammographictraining images each including a plurality of pixel intensity valuechannels for each pixel of a plurality of pixels thereof, training afirst sub-classifier of the trained statistical classifier according toa whole of each of the plurality of multi channel 2D mammographictraining images and the corresponding positive or negative indication ofmalignancy, for computation of a first score indicative of likelihood ofmalignancy within the respective whole multi channel 2D mammographicimage, extracting a plurality of patches for each of the plurality ofmulti channel 2D mammographic images, training a second sub-classifierof the trained statistical classifier according to the plurality ofpatches of each of the plurality of multi channel 2D mammographic imagesand the corresponding positive or negative indication of malignancy andlocation of the malignancy, for computation of a respective second scoreof a plurality of second scores indicative of likelihood of malignancywithin each respective patch of the plurality of patches, training agating sub-classifier of the trained statistical classifier according tothe first score and the plurality of second scores, the correspondingindication of likelihood of malignancy, and the location of themalignancy, and providing the trained statistical classifier includingthe first sub-classifier, the second sub-classifier, and the gatingsub-classifier, for computation of likelihood of malignancy and locationof the malignancy within a new single channel 2D mammographic image.

It is noted that the conversion of the single channel image to the multichannel image (as described herein) improves rates of detection ofmalignancy. The improvement in malignancy detection may be obtained bythe automated systems, methods, and/or code instructions describedherein, and/or may be obtained manually by improving the ability of aradiologist to visually detect malignancy in the multi channel image incomparison to the single channel image, as described herein inadditional detail.

The systems, methods, and/or code instructions described herein relateto the technical problem of improving cancer detection and/or diagnosticaccuracy for two dimensional (2D) digital screening mammographic images(i.e., x-ray based imaging modality), for example in terms ofsensitivity (e.g., 0.91), specificity (e.g., 0.78), and/or area undercurve (AUC) values similar to those of expert radiologistinterpretations of digital mammography and comparable to those describedfor digital breast tomosynthesis as described in additional detail inthe Experiment section below, and for example, as described withreference to M. A. Helvie, “Digital Mammography Imaging: BreastTomosynthesis and Advanced Applications,” Radiologic Clinics of NorthAmerica, vol. 48, no. 5. pp. 917-929, 2010, E. A. Rafferty, J. M. Park,L. E. Philpotts, S. P. Poplack, J. H. Sumkin, E. F. Halpern, and L. T.Niklason, “Assessing radiologist performance using combined digitalmammography and breast tomosynthesis compared with digital mammographyalone: results of a multicenter, multireader trial.,” Radiology, vol.266, no. 1, pp. 104-13, January 2013, P. Skaane, A. I. Bandos, R.Gullien, E. B. Eben, U. Ekseth, U. Haakenaasen, M. Izadi, I. N. Jebsen,G. Jahr, M. Krager, L. T. Niklason, S. Hofvind, and D. Gur, “Comparisonof digital mammography alone and digital mammography plus tomosynthesisin a population-based screening program.,” Radiology, vol. 267, no. 1,pp. 47-56, April 2013.

The systems, methods, and/or code instructions described herein improveperformance of a computing unit that performs the automatic detection ofthe indication of malignancy in the 2D mammographic image. Theimprovement in performance may be based on an increase in accuracy,sensitivity, and/or specificity of detecting the indication ofmalignancy using existing computing resources (e.g., processor(s),and/or data storage), and/or improving the efficiency of detectingmalignancy by a reduction in processing time, a reduction in processorutilization, and/or a reduction in data storage requirements. Forexample the systems, methods, and/or code instructions described hereinmay detect an abnormality in the breast tissue, and classify theabnormality into benign or indicative of malignancy, for example, ratherthan detecting the abnormality and leaving the diagnosis of benign ormalignancy to the physician which may require a biopsy. In yet anotherexample, the implementation of the statistical classifier based on oneneural network that processes full images, and another neural networkthat processes image patches, improves computational performance of thecomputing device by providing for independent hyper-parameter alterationfor each of the respective image scales (i.e., full image and imagepatches), for example, by combining features of malignancy that are bestrevealed at the whole image level (e.g., regional architecturaldistortion, and/or asymmetry) with features that are best revealed atthe patch level (e.g., micro-calcification and/or masses).

In a further implementation form of the first, second, and thirdaspects, the single channel 2D mammographic image comprises a black andwhite image, and the multi channel 2D mammographic image comprises atwo, three, or four channel false color image.

In a further implementation form of the first, second, and thirdaspects, the converting is executed based on Contrast Limited AdaptiveHistogram Equalization (CLAHE) and/or variants thereof.

In a further implementation form of the first, second, and thirdaspects, variants of CLAHE include at least one of: 2 layers, 3 layers,4 layers, and SPCLAHE.

In a further implementation form of the first, second, and thirdaspects, the CLAHE is applied to the single channel image by spreadingof parameters across the available channels of the multi channel image.

In a further implementation form of the first, second, and thirdaspects, the CLAHE is applied according to the following: high clippingvalue and low resolution window size on a first channel is fed to a redchannel of the multi channel 2D mammographic image, mid clipping valueand mid resolution window size on a second channel is fed to a greenchannel of the multi channel 2D mammographic image, and low clippingvalue and high resolution window size on a third channel is fed to the achannel of the multi channel 2D mammographic image.

In a further implementation form of the first, second, and thirdaspects, high clipping value is 16, low resolution window size is 4, midclipping value is 4, mid resolution window size is 8, low clipping valueis 2, and high resolution windows size is 16.

In a further implementation form of the first, second, and thirdaspects, the method further comprises and/or the system further includescode for presenting on a display, the computed multi channelmammographic image, and a visual marking indicative of the location ofmalignancy.

In a further implementation form of the first, second, and thirdaspects, the converting is performed by: computing a plurality of setsof intensity histogram for each pixel of the plurality of pixels of thesingle channel 2D mammographic image, wherein each set of intensityhistograms is computed for each respective pixel according to aneighborhood of the respective pixel, wherein each member of each set ofintensity histograms corresponds to a respective pixel intensity valuechannel of the plurality of pixel intensity value channels, cutting offthe respective histogram of each member of each set of the intensityhistograms at a respective predefined threshold to compute a respectiveadaptive histogram, computing a respective transformation functionaccording to each respective adaptive histogram, and computing, for eachpixel, the plurality of pixel intensity value channels according torespective transformation functions corresponding to each respectivepixel intensity value channel.

In a further implementation form of the first, second, and thirdaspects, the respective predefined threshold for the adaptive histogramcorresponding to a red color channel of the plurality of channels islow, wherein the respective predefined threshold for the adaptivehistogram corresponding to a green color channel of the plurality ofcolor channels is intermediate, and wherein the respective predefinedthreshold for the adaptive histogram corresponding to a blue colorchannel of the plurality of channels is high.

In a further implementation form of the first, second, and thirdaspects, the method further comprises and/or the system further includescode for equally redistributing the cut-off portion of each respectivehistogram among histogram bins prior to computing the respectivetransformation function.

In a further implementation form of the first, second, and thirdaspects, the respective transformation function is computed according toa respective cumulating distribution function computed according to eachrespective adaptive histogram.

In a further implementation form of the first, second, and thirdaspects, the first sub-classifier and the second sub-classifier areimplemented as respective deep convolutional neural networks executed inparallel.

In a further implementation form of the first, second, and thirdaspects, the second sub-classifier is implemented as a FasterRCNN, andwherein dimensions of each respective patch of the plurality of patchesare variable and are dynamically computed by a region proposal network(RPN).

In a further implementation form of the first, second, and thirdaspects, the gating sub-classifier comprises a random forest gatingnetwork.

In a further implementation form of the first, second, and thirdaspects, the gating sub-classifier computes at least one patchassociated with the indication of likelihood of malignancy, wherein thelocation of the at least one patch within the image corresponds to thelocation of the malignancy, wherein the at least one patch is computedbased on reduction of candidate patches according to non-maximalsuppression (NMS).

In a further implementation form of the first, second, and thirdaspects, single channel 2D mammographic images associated with theindication of relatively low likelihood of malignancy are furthercategorized as benign, indicative of breast tissue with at least onebenign abnormality, or categorized as normal, indicative of normalbreast tissue.

In a further implementation form of the first, second, and thirdaspects, the plurality of patches are computed by a sliding windowregion of interest (ROI).

In a further implementation form of the first, second, and thirdaspects, the method further comprises and/or the system further includescode for down-sampling the pixel resolution of each whole multi channel2D mammographic training image according to the input dimensions of thefirst sub-classifier and down-sampling the pixel resolution of each ofthe plurality of patches according to input dimensions of the secondsub-classifier.

In a further implementation form of the first, second, and thirdaspects, the first sub-classifier and the second sub-classifier areimplemented as respective deep convolutional neural networks (CNN),wherein each respective deep CNN is trained according to transferlearning based on features learned by lower layers of a pre-trainednetwork, while fine tuning existing snapshots.

In a further implementation form of the first, second, and thirdaspects, the trained statistical classifier computes, for a new singlechannel 2D mammographic image, one of two classification categoriesindicative of suspicious for malignancy or indicative of non-suspiciousfor malignancy.

In a further implementation form of the first, second, and thirdaspects, the first sub-classifier and the second sub-classifier areimplemented as respective deep CNN, and the gating sub-classifier isimplemented as a random forest classifier.

In a further implementation form of the first, second, and thirdaspects, the random forest classifier is set according to one or more ofthe following parameter values: max_depth=5, n_estimators=46,max_features=33, and random_state=1.

In a further implementation form of the third aspect, training thesecond sub-classifier, identifying a certain patch associated with anindication of malignancy as a hard negative finding when the secondsub-classifier incorrectly identifies the certain patch as indicative ofa low likelihood of malignancy, and re-training the secondsub-classifier according to the certain patch identified as the hardnegative finding.

In a further implementation form of the third aspect, each respectivetraining image of the plurality of single channel 2D mammographictraining images is associated with additional data comprising at leastone of: breast density of the breast appearing in the respectivetraining image, age of the target individual associated with therespective training image, and type of breast tissue abnormality presentin the respective training image, and wherein at least one of: the firstsub-classifier, the second-sub classifier, and the gatingsub-classifier, are trained according to the additional data forcomputation of likelihood of malignancy and location of the malignancywithin the new single channel 2D mammographic image of a targetindividual in associated with additional data of the target individual.

In a further implementation form of the second aspect, the systemfurther comprises code for training the statistical classifier, the codecomprising: code for receiving a plurality of single channel 2Dmammographic training images, each 2D mammographic training image of theplurality of 2D mammographic training images including of at least aportion of a breast, wherein each single channel 2D mammographictraining image includes a single pixel intensity value for each pixel ofa plurality of pixels thereof, wherein each single channel 2Dmammographic training image of the plurality of single channel 2Dmammographic training images is associated with a positive indication ofmalignancy or a negative indication of malignancy, wherein each memberof a sub-set of the plurality of single channel 2D mammographic trainingimages associated with the positive indication of malignancy is furtherassociated with a location of the malignancy, code for converting theplurality of single channel 2D mammographic training images intocorresponding plurality of multi channel 2D mammographic training imageseach including a plurality of pixel intensity value channels for eachpixel of a plurality of pixels thereof, code for training a firstsub-classifier of the trained statistical classifier according to awhole of each of the plurality of multi channel 2D mammographic trainingimages and the corresponding positive or negative indication ofmalignancy, for computation of a first score indicative of likelihood ofmalignancy within the respective whole multi channel 2D mammographicimage, code for extracting a plurality of patches for each of theplurality of multi channel 2D mammographic images, code for training asecond sub-classifier of the trained statistical classifier according tothe plurality of patches of each of the plurality of multi channel 2Dmammographic images and the corresponding positive or negativeindication of malignancy and location of the malignancy, for computationof a respective second score of a plurality of second scores indicativeof likelihood of malignancy within each respective patch of theplurality of patches, code for training a gating sub-classifier of thetrained statistical classifier according to the first score and theplurality of second scores, the corresponding indication of likelihoodof malignancy, and the location of the malignancy, and code forproviding the trained statistical classifier including the firstsub-classifier, the second sub-classifier, and the gatingsub-classifier, for computation of likelihood of malignancy and locationof the malignancy within a new single channel 2D mammographic image.

Unless otherwise defined, all technical and/or scientific terms usedherein have the same meaning as commonly understood by one of ordinaryskill in the art to which the invention pertains. Although methods andmaterials similar or equivalent to those described herein can be used inthe practice or testing of embodiments of the invention, exemplarymethods and/or materials are described below. In case of conflict, thepatent specification, including definitions, will control. In addition,the materials, methods, and examples are illustrative only and are notintended to be necessarily limiting.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

Some embodiments of the invention are herein described, by way ofexample only, with reference to the accompanying drawings. With specificreference now to the drawings in detail, it is stressed that theparticulars shown are by way of example and for purposes of illustrativediscussion of embodiments of the invention. In this regard, thedescription taken with the drawings makes apparent to those skilled inthe art how embodiments of the invention may be practiced.

In the drawings:

FIG. 1 is a flowchart of a method of converting a single channel 2Dmammographic image into a multi channel image and detecting anindication of malignancy by a trained statistical classifier comprisinga first neural network that analyzes the whole image, a second neuralnetwork that analyzes patches of the image, and a gating component thatreceives the outputs of the first and second networks and computes theindication of malignancy, in accordance with some embodiments of thepresent invention;

FIG. 2 is a block diagram of components of a system for converting asingle channel 2D mammographic image into a multi channel image anddetecting an indication of malignancy according to the multi channelimage by a trained statistical classifier comprising a first neuralnetwork that analyzes the whole image, a second neural network thatanalyzes patches of the image, and a gating component that receives theoutputs of the first and second networks and computes the indication ofmalignancy, in accordance with some embodiments of the presentinvention;

FIG. 3 is a flowchart of a method of training the statisticalclassifier, in accordance with some embodiments of the presentinvention;

FIG. 4 includes examples of multi channel mammographic images computedfrom single channel mammographic images, in accordance with someembodiments of the present invention;

FIGS. 5A-5B include examples of a single channel mammographic image andmulti channel mammographic image(s) that are computed from the singlechannel mammographic image, in accordance with some embodiments of thepresent invention;

FIG. 6 is a dataflow diagram depicting computation of the indication ofmalignancy for a mammographic image by a trained statistical classifier,in accordance with some embodiments of the present invention; and

FIG. 7 is a receiver operating curve (ROC) depicting the performanceresults obtained for the computational experiment described herein.

DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION

The present invention, in some embodiments thereof, relates tomammographic image processing and, more specifically, but notexclusively, to systems and methods for automated detection ofmalignancy in a mammographic image.

An aspect of some embodiments of the present invention relates tosystems, an apparatus, methods, and/or code instructions (stored in adata storage device, executable by one or more hardware processors) forcomputing an indication of likelihood of malignancy and optionally thelocation of the malignancy, of a two dimensional (2D) x-ray baseddigital single channel mammographic image by a trained statisticalclassifier. The single channel 2D mammographic image, which includes asingle intensity value for each pixel, is converted into a multi channel2D mammographic image, which includes multiple intensity values for eachpixel. There may be, for example, 2, 3, 4, or more channels per pixel.For example, a black and white image is converted into a false colorimage based on the red, green, and blue color channels. The conversionfrom single channel 2D mammographic image into the multi channel 2Dmammographic image may be executed based on Contrast Limited AdaptiveHistogram Equalization (CLAHE) and/or variants thereof, for example, 2,3, 4 layers, and/or SPCLAHE. A first sub-classifier (optionally a deepconvolutional neural network (CNN)) computes a first score indicative oflikelihood of malignancy within the whole multi channel 2D mammographicimage. A second sub-classifier (optionally another deep CNN, or a threestate network based on FasterRCNN and/or Yolo) computes a second scorefor each of multiple patches extracted from the multi channel 2Dmammographic image. Patches may be fixed in size, with a constant fixedsize for the patches (e.g., all patches), and/or the size of the matchesmay be variable and/or dynamically computed by a region proposal network(RPN). The second score is indicative of likelihood of malignancy withinthe respective patch. The first and second sub-classifiers may beexecuted in parallel. A gating sub-classifier (optionally a randomforest classifier, and/or based on reduction of candidate patches toidentify patch(es) including the indication of likelihood of malignancyaccording to non-maximal suppression (NMS)) receives the computed firstscore and second scores, and computes an indication of likelihood ofmalignancy, and optionally the location of the malignancy according tothe location of the image corresponding to one or more extractedpatches.

Optionally, the conversion from single channel 2D mammographic imageinto the multi channel 2D mammographic image is executed based on CLAHEand/or variants thereof, for example, 2, 3, 4 layers, and/or SPCLAHE.Optionally, the CLAHE is applied to the single channel image, byspreading of parameters across the available channels of the multichannel output image. It is noted that the application of CLAHE forconversion of a single channel image into a multi channel image asdescribed herein is in contrast to the standard application of CLAHE,which traditionally maintains the single channel, for example, maintainsa grayscale image as grayscale without performing conversion asdescribed herein, is applied only to the luminance channel of an HLSimage without performing conversion as described herein, and/or appliesthe same parameters for each channel of an already existing RGB imagewithout performing conversion as described herein.

It is noted that the conversion of the single channel image to the multichannel image (as described herein) improves rates of detection ofmalignancy. The improvement in malignancy detection may be obtained bythe automated systems, methods, and/or code instructions describedherein, and/or may be obtained manually by improving the ability of aradiologist to visually detect malignancy in the multi channel image incomparison to the single channel image, as described herein inadditional detail.

The systems, methods, and/or code instructions described herein relateto the technical problem of improving cancer detection and/or diagnosticaccuracy for two dimensional (2D) digital screening mammographic images(i.e., x-ray based imaging modality), for example in terms ofsensitivity (e.g., 0.91), specificity (e.g., 0.78), and/or area undercurve (AUC) values similar to those of expert radiologistinterpretations of digital mammography and comparable to those describedfor digital breast tomosynthesis as described in additional detail inthe Experiment section below, and for example, as described withreference to M. A. Helvie, “Digital Mammography Imaging: BreastTomosynthesis and Advanced Applications,” Radiologic Clinics of NorthAmerica, vol. 48, no. 5. pp. 917-929, 2010, E. A. Rafferty, J. M. Park,L. E. Philpotts, S. P. Poplack, J. H. Sumkin, E. F. Halpern, and L. T.Niklason, “Assessing radiologist performance using combined digitalmammography and breast tomosynthesis compared with digital mammographyalone: results of a multicenter, multireader trial.,” Radiology, vol.266, no. 1, pp. 104-13, January 2013, P. Skaane, A. I. Bandos, R.Gullien, E. B. Eben, U. Ekseth, U. Haakenaasen, M. Izadi, I. N. Jebsen,G. Jahr, M. Krager, L. T. Niklason, S. Hofvind, and D. Gur, “Comparisonof digital mammography alone and digital mammography plus tomosynthesisin a population-based screening program.,” Radiology, vol. 267, no. 1,pp. 47-56, April 2013.

In particular, the technical problem may relate to differentiatingbetween abnormal breast tissue that is benign, and abnormal breasttissue that is (suspicious for) malignant.

Computer Aided Detection (CAD) for mammography was first approved by theFood and Drug Administration (FDA) in 1998. CAD software functionsessentially as a “second reader” to the interpreting radiologist. Earlystudies demonstrated increases of 19-23% in breast cancer detection ratewith CAD utilization, resulting in reimbursement qualification andwidespread adoption in the US, for example, as described with referenceto T. W. Freer and M. J. Ulissey, “Screening mammography withcomputer-aided detection: prospective study of 12,860 patients in acommunity breast center.,” Radiology, vol. 220, no. 3, pp. 781-6, 2001,J. J. Fenton, G. Xing, J. G. Elmore, H. Bang, S. L. Chen, K. K.Lindfors, and L. M. Baldwin, “Short-term outcomes of screeningmammography using computer-aided detection a population-based study ofmedicare enrollees,” Ann. Intern. Med., vol. 158, no. 8, pp. 580-587,2013, and V. M. Rao, D. C. Levin, L. Parker, B. Cavanaugh, A. J.Frangos, and J. H. Sunshine, “How widely is computer-aided detectionused in screening and diagnostic mammography?,” J. Am. Coll. Radiol.,vol. 7, no. 10, pp. 802-805, 2010. However, despite subsequent upgradesin traditional CAD algorithms, its clinical utility has remainedcontroversial. The most definitive study to date pooled data frommammography registries of over 500,000 mammograms performed between2003-2009 and found no added benefit of CAD in cancer detection ordiagnostic accuracy for screening mammography, for example, as describedwith reference to C. D. Lehman, R. D. Wellman, D. S. M. Buist, K.Kerlikowske, A. N. A. Tosteson, D. L. Miglioretti, and Breast CancerSurveillance Consortium, “Diagnostic Accuracy of Digital ScreeningMammography With and Without Computer-Aided Detection.,” JAMA Intern.Med., vol. 175, no. 11, pp. 1828-37, November 2015. Traditional CADalgorithms deploy conventional computer vision technologies based upondetection of hand-crafted imaging features broadly categorized intomasses or micro-calcifications, which have not been sufficientlyeffective to improve cancer detection and/or cancer diagnostic accuracyin screening mammograms. Digital mammography is the foundation of breastimaging practice and the only imaging modality to demonstrate mortalityreduction with screening program, for example, as described withreference to L. Tabar, B. Vitak, T. Chen, A. Yen, A. Cohen, T. Tot, S.Chiu, S. Chen, J. Fann, J. Rosell, H. Fohlin, R. Smith, S. Duffy, and E.Al, “Swedish two-county trial: impact of mammographic screening onbreast cancer mortality during 3 decades—with comments,” Radiology, vol.260, no. 3, pp. 658-663, 2011, and B. Lauby-Secretan, C. Scoccianti, D.Loomis, L. Benbrahim-Tallaa, V. Bouvard, F. Bianchini, and K. Straif,“Breast-cancer screening—viewpoint of the IARC Working Group.,” N. Engl.J. Med., vol. 372, no. 24, pp. 2353-8, 2015. However, mammographycontinues to underperform with variable sensitivity and specificity,even with widespread CAD implementation, for example, as described withreference to A. M. Kavanagh, G. G. Giles, H. Mitchell, and J. N. Cawson,“The sensitivity, specificity, and positive predictive value ofscreening mammography and symptomatic status.,” J. Med. Screen., vol. 7,no. 2, pp. 105-10, 2000, C. D. Lehman, R. D. Wellman, D. S. M. Buist, K.Kerlikowske, A. N. A. Tosteson, D. L. Miglioretti, and Breast CancerSurveillance Consortium, “Diagnostic Accuracy of Digital ScreeningMammography With and Without Computer-Aided Detection.,” JAMA Intern.Med., vol. 175, no. 11, pp. 1828-37, November 2015, N. S. Winkler, S.Raza, M. Mackesy, and R. L. Birdwell, “Breast density: clinicalimplications and assessment methods.,” Radiographics, vol. 35, no. 2,pp. 316-24, 2015, T. M. Kolb, J. Lichy, and J. H. Newhouse, “Comparisonof the Performance of Screening Mammography, Physical Examination, andBreast US and Evaluation of Factors that Influence Them: An Analysis of27,825 Patient Evaluations,” Radiology, vol. 225, no. 1, pp. 165-175,October 2002, and K. Kerlikowske, R. A. Hubbard, D. L. Miglioretti, B.M. Geller, B. C. Yankaskas, C. D. Lehman, S. H. Taplin, and E. A.Sickles, “Comparative effectiveness of digital versus film-screenmammography in community practice in the United States: A cohort study,”Ann. Intern. Med., vol. 155, no. 8, pp. 493-502, 2011.

The systems, methods, and/or code instructions described herein improveperformance of a computing unit that performs the automatic detection ofthe indication of malignancy in the 2D mammographic image. Theimprovement in performance may be based on an increase in accuracy,sensitivity, and/or specificity of detecting the indication ofmalignancy using existing computing resources (e.g., processor(s),and/or data storage), and/or improving the efficiency of detectingmalignancy by a reduction in processing time, a reduction in processorutilization, and/or a reduction in data storage requirements. Forexample the systems, methods, and/or code instructions described hereinmay detect an abnormality in the breast tissue, and classify theabnormality into benign or indicative of malignancy, for example, ratherthan detecting the abnormality and leaving the diagnosis of benign ormalignancy to the physician which may require a biopsy. In yet anotherexample, the implementation of the statistical classifier based on oneneural network that processes full images, and another neural networkthat processes image patches, improves computational performance of thecomputing device by providing for independent hyper-parameter alterationfor each of the respective image scales (i.e., full image and imagepatches), for example, by combining features of malignancy that are bestrevealed at the whole image level (e.g., regional architecturaldistortion, and/or asymmetry) with features that are best revealed atthe patch level (e.g., micro-calcification and/or masses).

The systems, methods, and/or code instructions described herein improvean underling technical process within the technical field of medicalimage processing, in particular, within the field of automatic analysisof 2D mammographic images to identify indications of breast cancer.

The systems, methods, and/or code instructions described herein providea unique, particular, and advanced technique of analyzing 2Dmammographic images for detection of malignancy, by converting a singlechannel 2D mammographic image (e.g., black and white) into a multichannel image (e.g., false color based on red, green, and blue), and/orby applying a trained classifier that includes a first neural networktrained to classify the full multi channel image a second neural networkthat classifies patches of the multi channel image and a gatingcomponent that processes the output of the first and second networks tocompute the indication of malignancy and optionally the location of theindication within the mammographic image.

The systems, methods, and/or code instructions described herein generatenew data in the form of the multi channel image (e.g., false colorenhanced image) computed from the 2D mammographic image, and/or generatenew data in the form of the trained classifier that includes the firstand second neural networks and the gating component.

The systems, methods, and/or code instructions described herein are tiedto physical real-life components, for example, x-ray machines thatgenerate the 2D digital mammographic image, and computational hardware(e.g., processors, physical memory devices) that analyze themammographic image.

Accordingly, the systems, methods, and/or code instructions describedherein are inextricably tied to computer technology and/or physicalcomponents (e.g., mammogram machine, processor(s), storage device(s)) toovercome an actual technical problem arising in processing and/oranalysis of 2D mammographic images.

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not necessarily limited in itsapplication to the details of construction and the arrangement of thecomponents and/or methods set forth in the following description and/orillustrated in the drawings and/or the Examples. The invention iscapable of other embodiments or of being practiced or carried out invarious ways.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, and any suitable combination of theforegoing. A computer readable storage medium, as used herein, is not tobe construed as being transitory signals per se, such as radio waves orother freely propagating electromagnetic waves, electromagnetic wavespropagating through a waveguide or other transmission media (e.g., lightpulses passing through a fiber-optic cable), or electrical signalstransmitted through a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Reference is now made to FIG. 1, which is a flowchart of a method ofconverting a single channel 2D mammographic image into a multi channelimage and detecting an indication of malignancy according to the multichannel image by a trained statistical classifier comprising a firstneural network that analyzes the whole image, a second neural networkthat analyzes patches of the image, and a gating component that receivesthe outputs of the first and second networks and computes the indicationof malignancy, in accordance with some embodiments of the presentinvention. Reference is also made to FIG. 2, which is a block diagram ofcomponents of a system 200 for converting a single channel 2Dmammographic image into a multi channel image and detecting anindication of malignancy according to the multi channel image by atrained statistical classifier comprising a first neural network thatanalyzes the whole image, a second neural network that analyzes patchesof the image, and a gating component that receives the outputs of thefirst and second networks and computes the indication of malignancy, inaccordance with some embodiments of the present invention. Reference isalso made to FIG. 3, which is a flowchart of a method of training thestatistical classifier described with reference to FIG. 1, in accordancewith some embodiments of the present invention. System 200 may implementthe acts of the method described with reference to FIG. 1 and/or FIG. 3,optionally by a hardware processor(s) 202 of a computing device 204executing code instructions stored in a data storage 206.

Computing unit 204 may be implemented as, for example, a clientterminal, a server, a radiology workstation, a virtual machine, acomputing cloud, a mobile device, a desktop computer, a thin client, aSmartphone, a Tablet computer, a laptop computer, a wearable computer,glasses computer, and a watch computer.

Computing unit 204 may include locally stored software that performs oneor more of the acts described with reference to FIG. 1 and/or FIG. 3,and/or may act as one or more servers (e.g., network server, web server,a computing cloud, virtual server) that provides services (e.g., one ormore of the acts described with reference to FIG. 1 and/or FIG. 3) toone or more client terminals 208 (e.g., remotely located radiologyworkstations) over a network 210, for example, providing software as aservice (SaaS) to the client terminal(s) 208, providing an applicationfor local download to the client terminal(s) 208, and/or providingfunctions using a remote access session to the client terminals 208,such as through a web browser.

Computing unit 204 receives 2D mammographic image(s) captured by amammogram machine(s) 212, for example, a standard x-ray based 2Dmammographic imaging device for performing screening mammograms.Mammographic images captured by mammogram machine 212 may be stored in amammogram repository 214, for example, a storage server, a computingcloud, virtual memory, and a hard disk. The mammographic images storedby mammogram repository 214 may include mammogram images of patients foranalysis, and/or training images 216 that have been previously analyzed(e.g., by radiologists) and labeled with findings indicative ofmalignancy or no malignancy.

Training images 216 are used to train the statistical classifier, asdescribed herein. It is noted that training images 216 may be stored bya server 218, accessibly by computing unit 204 over network 210, forexample, a publicly available training dataset, and/or a customizedtraining dataset created for training the statistical classifierdescribed herein.

Computing unit 204 may receive the mammographic image(s) from mammogramdevice 212 and/or mammogram repository 214 using one or more imaginginterfaces 220, for example, a wire connection (e.g., physical port), awireless connection (e.g., antenna), a local bus, a port for connectionof a data storage device, a network interface card, other physicalinterface implementations, and/or virtual interfaces (e.g., softwareinterface, virtual private network (VPN) connection, applicationprogramming interface (API), software development kit (SDK)).

Hardware processor(s) 202 may be implemented, for example, as a centralprocessing unit(s) (CPU), a graphics processing unit(s) (GPU), fieldprogrammable gate array(s) (FPGA), digital signal processor(s) (DSP),and application specific integrated circuit(s) (ASIC). Processor(s) 204may include one or more processors (homogenous or heterogeneous), whichmay be arranged for parallel processing, as clusters and/or as one ormore multi core processing units.

Data storage device 206 (also referred to herein as a program store,and/or memory) stored code instruction for execution by hardwareprocessor(s) 202, for example, a random access memory (RAM), read-onlymemory (ROM), and/or a storage device, for example, non-volatile memory,magnetic media, semiconductor memory devices, hard drive, removablestorage, and optical media (e.g., DVD, CD-ROM). For example, programstore 206 may store image processing code 206A that implement one ormore acts and/or features of the method described with reference to FIG.1, and/or training code 206B that execute one or more acts of the methoddescribed with reference to FIG. 3.

Computing device 204 may include a data repository device 222 forstoring data, for example, a trained classifier 222A (as describedherein), training images 216, and/or electronic medical records. Datarepository device 222 may be implemented as, for example, a memory, alocal hard-drive, a removable storage unit, an optical disk, a storagedevice, and/or as a remote server and/or computing cloud (e.g., accessedover network 210). It is noted that trained classifier 222A, trainingimages 216, and/or electronic medical records may be stored in datarepository device 222, with executing portions loaded into data storagedevice 206 for execution by processor(s) 202.

Computing device 204 may include data interface 224, optionally anetwork interface, for connecting to network 210, for example, one ormore of, a network interface card, a wireless interface to connect to awireless network, a physical interface for connecting to a cable fornetwork connectivity, a virtual interface implemented in software,network communication software providing higher layers of networkconnectivity, and/or other implementations. Computing device 204 mayaccess one or more remote servers 218 using network 210, for example, todownload updated training images 216 and/or to download an updatedversion of image processing code, training code, and/or the trainedclassifier.

Computing device 204 may communicate using network 210 (or anothercommunication channel, such as through a direct link (e.g., cable,wireless) and/or indirect link (e.g., via an intermediary computing unitsuch as a server, and/or via a storage device) with one or more of:

-   -   Client terminal(s) 208, for example, when computing device 204        acts as a server providing image analysis services (e.g., SaaS)        to remote radiology terminals, for analyzing remotely obtained        mammographic images for detection of indication of malignancy.    -   Server 218, for example, implemented in association with a        picture archiving and communication system (PACS), which may        storage large numbers of mammographic images for analysis, for        example, captured by a mammographic machine of a radiology        clinic.    -   Mammogram repository 214 that stores mammographic images and/or        mammogram device 212 that outputs the digital 2D mammographic        image(s).

It is noted that imaging interface 220 and data interface 224 may existas two independent interfaces (e.g., two network ports), as two virtualinterfaces on a common physical interface (e.g., virtual networks on acommon network port), and/or integrated into a single interface (e.g.,network interface).

Computing device 204 includes or is in communication with a userinterface 226 allowing a user to enter data and/or view the computedmalignancy indication, view the location of the indication ofmalignancy, view the mammographic image, and/or view the false colorenhanced converted image. Exemplary user interfaces 226 include, forexample, one or more of, a touchscreen, a display, a keyboard, a mouse,and voice activated software using speakers and microphone.

Referring now to FIG. 1, at 102, the statistical classifier (e.g., 220A)for computation of the indication of likelihood of malignancy andoptionally computation of an indication of localization of themalignancy is trained, as described in additional detail below withreference to FIG. 3.

At 104, a digital single channel two dimensional (2D) x-ray basedmammographic image of at least a portion of a breast is received bycomputing device 204. The digital single channel 2D mammographic imageincludes a single pixel intensity value for each pixel, for example, ona scale of 0-255, or other scales. The single channel 2D image may bepresented as a black a white image, where the shade is defined accordingto the intensity values of the pixels.

The digital single channel 2D mammographic image may be obtained as a(e.g., pseudo) slice from a 3D mammographic imaging system, for example,a tomosynthesis system.

The 2D mammographic image may be obtained by mammogram machine 212, forexample, as part of a routine breast cancer screening mammogram. The 2Dmammographic image may be stored in mammogram repository 214 (e.g., ahard drive of mammogram machine 212, a PACS server, a CD-ROM diskprovided to the patient) and provided to computing device 204 usingimaging interface 218 (e.g., network connection, CD-ROM drive, cableconnection, local bus, port for connecting a data storage device).

Optionally, additional data is obtained, for example, as described withreference to act 304 of FIG. 3. The additional data may be obtained, forexample, from an electronic health and/or medical record (e.g., storedin a database) of the target individual for which the mammographic imageis captured. The additional data may improve accuracy of the statisticalclassifier.

At 106, the single channel 2D mammographic image is converted into amulti channel 2D mammographic image, for example, 2, 3, 4, or greaternumber of channels per pixel. The multi channel image including multiplepixel intensity value channels for each pixel. The multi channel imagemay be presented as a false color image, where each pixel intensityvalue channel denotes a certain color channel. For example, each singlechannel of each pixel of the 2D mammographic image is converted intothree channels denoting red, green, and blue channels of a color image.

The conversion from single to multi channel may be performed for each(optional pseudo) slice obtained from a 3D mammographic imaging system(e.g., tomosynthesis). Each slice is converted as described herein.

The conversion from the single channel 2D mammographic image (e.g.,black and white) into the multi channel 2D mammographic image (e.g.,false color) may be implemented based on CLAHE and/or variationsthereof, for example, 2, 3, 4 layers and/or SPCLAHE.

Optionally, the CLAHE is applied to the single channel image, byspreading of parameters across the available channels of the multichannel output. Optionally, for a received single channel (e.g.,grayscale) image, CLAHE is applied to three (or other number, forexample, 2, or 4) instances of the single channel image, for example,with the following:

-   -   High clipping value (e.g. 16 or other value) and low resolution        window size (e.g., 4, or other value) on the first channel,        which is fed to the red channel of the resultant processed        image.    -   Mid clipping value (e.g., 4, or other value) and mid resolution        window size (e.g., 8, or other value) on the second channel,        which is fed to the green channel of the resultant processed        image.    -   Low clipping value (e.g., 2, or other value) and high resolution        window size (e.g., 16, or other value) on the third channel,        which is fed to the blue channel of the resultant processed        image.

It is noted that the color channels may be implemented in anothercombination.

The conversion (e.g., based on CLAHE and/or variants thereof) may beperformed, for example, according to one or more of the followingexemplary features and/or variations thereof:

-   -   A set of intensity histograms is computed for respective pixels        of the single channel 2D mammographic image. Each set of        intensity histograms is computed for each respective pixel        according to a neighborhood of the respective pixel. The size of        the neighborhood may be defined according to a requirement. Each        member intensity histogram of each set of intensity histograms        corresponds to a respective pixel intensity value channel of        multiple pixel intensity value channels of the multi channel        mammographic image. For example, each intensity histogram        denotes a respective false color channel of a computed false        color mammographic image.    -   Each member intensity histogram of each set of intensity        histograms is cut off at a respective predefined threshold to        compute a corresponding respective adaptive histogram. For        example, intensity histograms of pixels corresponding to each        certain false color are cut off at corresponding predefined        value. For example, intensity histograms of pixels corresponding        to the red false color channel are cut off at one predefined        value, intensity histograms of pixels corresponding to the green        false color channel are cut off at another predefined value, and        intensity histograms of pixels corresponding to the green false        color channel are cut off at another predefined value.

Optionally, the respective predefined threshold for the adaptivehistogram corresponding to the red color channel is low, the respectivepredefined threshold for the adaptive histogram corresponding to thegreen color channel is intermediate, and the respective predefinedthreshold for the adaptive histogram corresponding to the blue colorchannel is high. It is noted that other implementations may be selected,for example, different color channels for the low, medium and highthresholds. It is noted that other color spaces may be selected, forexample, cyan, yellow, and magenta.

-   -   A respective transformation function is computed according to        each respective adaptive histogram. The respective        transformation function may be computed according to a        respective cumulating distribution function which is computed        according to each respective adaptive histogram.    -   The cut-off portion of each respective histogram (which is        defined according to the respective predefined threshold) is        equally redistributing among the histogram bins prior to        computing the respective transformation function.    -   The respective intensity value of each of the multiple pixel        intensity value channels of each pixel of the image is computed        according to the transformation function corresponding to the        respective pixel intensity value channel, for example, the        intensity value of the red channel is computed according to the        transformation function computed for the red channel.

At 108, a first sub-classifier of the trained statistical classifier(e.g., 220A) computes, according to the whole multi channel 2Dmammographic image, a first score indicative of likelihood of malignancywithin the whole multi channel 2D mammographic image.

As used herein, the term whole may refer to the entire image, or themajority of the image, or a significant portion of the image. The termwhole refers to a portion of the image that is larger than the extractedpatches described herein. The term whole refers to a single analysis ofthe multi channel 2D mammographic image by the first sub-classifier.

Optionally, the pixel resolution of the whole multi channel 2Dmammographic image is down-sampled according to the input dimensions ofthe first sub-classifier. For example, when the single channel 2Dmammographic image is captured at a higher resolution that theimplementation of the first sub-classifier, the multi channel 2Dmammographic image is down sampled according to the input implementationof the first sub-classifier.

The first score is indicative of the likelihood of malignancy presentsomewhere within the whole image. The first score may be implemented as,for example, a probability of the presence of malignancy, and/or abinary classification indicative of likelihood of malignancy orlikelihood of no malignancy. The first score may be further indicativeof likelihood of no malignancy within a detected abnormality, forexample, abnormal breast tissue which is benign.

Optionally, the first sub-classifier is implemented as a deepconvolutional neural network.

At 110, a second sub-classifier of the trained statistical classifiercomputes, according to each respective patch of multiple patchesextracted from the multi channel 2D mammographic image, a respectivesecond score of indicative of likelihood of malignancy within eachrespective patch. Multiple second scores are computed, for example, asingle or set of second scores for each patch.

Each second score is indicative of the likelihood of malignancy presentsomewhere within the corresponding patch. The second score may beimplemented as, for example, a probability of the presence ofmalignancy, and/or a binary classification indicative of likelihood ofmalignancy or likelihood of no malignancy. The second score may befurther indicative of likelihood of no malignancy within a detectedabnormality, for example, abnormal breast tissue which is benign.

It is noted that the second score is indicative of the location ofmalignancy within the whole image. For example, when a certain (ormultiple overlapping) patch is indicative of likelihood of malignancy,the location of malignancy within the whole image is denoted by thelocation of the patch within the whole image.

Patches are extracted from the multi channel 2D mammographic image priorto computation of the second score by the second sub-classifier. Patchesmay be computed by a sliding window region of interest (ROI) over themulti channel 2D mammographic image. Patches may overlap one another,and/or may be adjacent without overlapping.

Optionally, about 100-400 patches are extracted from each multi channel2D mammographic image.

Optionally, the dimensions of the extracted patches are predefined andconstant in size for all of the extracted patches. For example, thesliding window strides measure about ⅓ of the window's width.Alternatively or additionally, the dimensions of the patches arevariable. Optionally, the dimensions of extracted patches (e.g., eachpatch) are dynamically computed. The dynamic computation of the variablepatches may be performed by a third component, for example, by a regionproposal network (PRN), for example, a component of a FasterRCNN.

Optionally, the pixel resolution of each patch (or the pixel resolutionof the multi channel 2D mammographic image) is down-sampled according tothe input dimensions of the second sub-classifier. Down sampling may beperformed for each extracted patch, and/or for the multi channel 2Dmammographic image prior to patch extraction. For example, when thedimension of each patch is 500 pixels×500 pixels, the patch may bedown-sampled to obtain 299 pixel×299 pixel patches (optionally with red,green, blue color enhancement denoted as ×3) for conforming to thenetwork input dimensions.

Optionally, the second sub-classifier is implemented as a deepconvolutional neural network.

Optionally, the first sub-classifier and the second sub-classifier areexecuted in parallel. The parallel execution reduces processing time forcomputation of the overall classification by the statistical classifier.It is noted that the first and second sub-classifier may be executedsequentially, for example, in computational systems that are unable toperform parallel processing.

At 112, a gating sub-classifier of the trained statistical classifiercomputes, according to the first score and the second scores, anindication of likelihood of malignancy for the single channel 2Dmammographic image. Alternatively or additionally, the gatingsub-classifier outputs the location of the identified malignancy withinthe image. Alternatively or additionally, the gating sub-classifieroutputs the computed probability of the likelihood of malignancy.

The gating sub-classifier may output a binary value indicative oflikelihood of malignancy (e.g., suspicious for malignancy, high risk ofmalignancy) and indicative of likelihood of no malignancy (e.g., notsuspicious for malignancy, low risk of malignancy). When the computedindication is of likelihood of no malignancy, an indication of whetherthe no malignancy is associated with normal or abnormal breast tissuemay be generated, as a sub classification category, or as a thirdclassification category.

Optionally, the gating sub-classifier is implemented as a random forestgating network. Alternatively or additionally, the gating sub-classifiercomputes one or more patches associated with the indication oflikelihood of malignancy. The location of the patch(es) within the imagecorrespond to the location of the malignancy. The patch(es) is computedbased on reduction of candidate patches according to non-maximalsuppression (NMS).

At 114, the computed indication of likelihood of malignancy (or nomalignancy) computed by the trained classifier for the single channel 2Dmammographic image is provided. When the computed indication is forlikelihood of malignancy, an indication of localization of themalignancy within one or more patches or other location methods (e.g.,arrow, coordinates, border) of the single channel 2D mammographic imageis provided. When the computed indication is for likelihood ofnon-malignancy, a sub-indication of whether abnormal benign tissue isdetected may be provided.

The indication computed by the trained statistical classifier may beprovided as, for example, presented on a display (e.g., within agraphical user interface (GUI), optionally in association with apresentation of the single channel and/or multi channel mammographicimage, stored in an electronic health and/or medical record of thepatient (e.g., as metadata and/or in a predefined field, optionally inassociation with the mammographic image), and/or forwarded to a remoteserver for further analysis, storage, forwarding, and/or processing.

Optionally, the multi channel mammographic image is presented on thedisplay, for example, for visual inspection by a radiologist. A visualindication representing the computed location of malignancy may bepresented in association with the multi channel image, for example, as aborder outlining the location of the detected malignancy. The computedlikelihood (e.g., probability) of the detected malignancy may bepresented on the display in association with the multi channelmammographic image, for example, as a numerical value. Optionally, thesingle channel mammographic image is presented in association with themulti channel image, for example, located adjacent to the multi channelimage. The radiologist may visually inspect and/or compare between thesingle and multi channel images, for example, to visually confirm theautomated detected malignancy.

Referring now back to act 102 of FIG. 1, the statistical classifier(e.g., 220A) is trained and/or updated with additional training imagesas described with reference to FIG. 3. The statistical classifier may betrained by processor(s) 202 of computing device 204 executing trainingcode 206B. The statistical classifier may be locally trained bycomputing device 204, and/or remotely trained by server(s) 218. Thetrained statistical classifier may be obtained by computing device 204from server(s) 218 over network 210, and locally stored for localexecution (e.g., as trained classifier 220A stored in data repository222).

At 302, single channel x-ray based 2D mammographic training images 216are received. Each single channel 2D mammographic training imageincludes at least a portion of a breast. Each single channel 2Dmammographic training image includes a single pixel intensity value foreach pixel thereof. The single channel 2D mammographic images may becaptured by mammogram device 212 for breast cancer screening. Singlechannel 2D mammographic training images 216 may be stored in mammogramrepository 216, which is locally associated with computing device 204and/or stored remotely on serer 218 (obtained by computing device 204over network 210).

Training images 216 may include a variation of mammographic breastdensity.

Exemplary training images 216, for example The Digital Database forScreening Mammography (DDSM), and the Zebra Mammography Dataset (ZMDS)are described below with reference to the Experiment section.

At 304, the single channel 2D mammographic training images are eachassociated with a positive indication of malignancy or a negativeindication of malignancy. Optionally, each member of a sub-set of thesingle channel 2D mammographic training images associated with thepositive indication of malignancy is further associated with a locationof the malignancy. Optionally, each member of the subset of the singlechannel 2D mammographic images associated with the indication oflikelihood of no malignancy (e.g., relatively low likelihood ofmalignancy) are further categorized as benign indicative of breasttissue with one or more abnormality that are benign, or categorized asnormal indicative of normal breast tissue.

The association may be implemented, for example, as metadata associatedwith each training image, as a tag associated with each training image,and/or as a value in a field associated with each training image storedin a database.

The designation of positive indication, negative indication, benignabnormal breast tissue, and/or normal breast tissue, may be mademanually, for example, by one or more expert radiologists, optionallybased on a biopsy and/or other investigations.

The designation of a distinct benign classification category providesfor computation of a tri-categorical classifier that differentiatesnormal from abnormal breast tissue, and classifies abnormalities asindicative of likelihood of malignancy or indicative of non-malignancy(i.e., benign abnormal tissue).

Optionally, single channel 2D mammographic training images areassociated with additional data. Exemplary additional data includes oneor more of: breast density of the breast appearing in the respectivetraining image, age of the target individual associated with therespective training image, and type of breast tissue abnormality presentin the respective training image. One or more of: the firstsub-classifier, the second-sub classifier, and the gatingsub-classifier, are trained according to the additional data forcomputation of likelihood of malignancy and location of the malignancywithin the new single channel 2D mammographic image of a targetindividual in associated with additional data of the target individual.The additional data may be stored, for example, in an electronic healthand/or medical record of the target individuals, as metadata associatedwith the images, as tags associated with images, and/or in a database.

Additional exemplary details for designation of the training images isdescribed below with reference to the Experiment section. For example,positive ground truth is defined based on biopsy proven pathology, andnegative samples are defined by biopsy proven benign tissue and/or atleast 2 years of stable imaging follow up.

At 306, the single channel 2D mammographic training images are convertedinto corresponding multi channel 2D mammographic training images asdescribed with reference to act 106 of FIG. 1.

At 308, the first sub-classifier of the statistical classifier istrained according to each whole multi channel 2D mammographic trainingimage, the corresponding positive or negative indication of malignancy,and optionally the indication of normal or abnormal benign tissue (whenavailable and/or when relevant). The first sub-classifier is trained tocompute the first score indicative of likelihood of malignancy within areceived new whole multi channel 2D mammographic image, as describedwith reference to act 108 of FIG. 1.

At 310, the second sub-classifier of the statistical classifier istrained according to patches extracted from each of the multi channel 2Dmammographic images, the corresponding positive or negative indicationof malignancy, and optionally the location of the malignancy. It isnoted that the explicit location of malignancy is not necessarilyrequired, as the presence of malignancy within a certain patch denotesthe location of malignancy within the image according to the location ofthe certain patch within the image.

The second sub-classifier is trained to compute the respective secondscore indicative of likelihood of malignancy within each respectivepatch, as described with reference to act 110 of FIG. 1.

The patches are extracted for each of the multi channel 2D mammographicimages and optionally down-sampled, as described with reference to act110 of FIG. 1.

The patches may be augmented with flip and/or rotation operations, forexample, providing an 8-fold augmentation of the training patches.

Optionally, a certain patch associated with an indication of malignancyis identified as a hard negative finding when the second sub-classifierincorrectly identifies the certain patch as indicative of a lowlikelihood of malignancy, for example, by outputting the second scoreindicative of the low likelihood of malignancy, and/or outputting thesecond score indicative of non-suspicious for malignancy. The secondsub-classifier is re-trained according to the certain patch identifiedas the hard negative finding, by re-inputting the hard negative findinginto the second sub-classifier in association with the correctedindication of likelihood of malignancy. Multiple rounds of hard negativemining may be performed, for example, 2 or more rounds. The hardnegative mining is designed to improve specificity of the classifier. Itis noted that specificity is particularly important, as hundreds ofinferences may occur in each series assessment.

Optionally, the first sub-classifier and the second sub-classifier areimplemented as respective deep convolutional neural networks (CNN). Eachrespective deep CNN may be trained according to transfer learning basedon features learned by lower layers of a pre-trained network, while finetuning existing snapshots. The first sub-classifier and the secondsub-classifier may be trained in parallel. The parallel training reducesthe computational time for computing the trained statistical classifier.

It is noted that in contrast to other machine vision methods that arebased on hand crafted features, for example, broadly categorized asmasses or micro-calcifications, the neural networks are based on featurediscovery within ground truth validated training images.

At 312, the gating sub-classifier of the statistical classifier istrained according to the first score outputted by the firstsub-classifier and the second scores outputted by the secondsub-classifier, the corresponding indication of likelihood ofmalignancy, and optionally the location of the malignancy within theimage. The gating sub-classifier may be trained according to commonstatistical attributes of the second scores (computed based on thepatches extracted by the sliding window).

Optionally, the gating sub-classifier is implemented as a random forestclassifier. An exemplary random forest sub-classifier is set accordingto one or more of the following parameter values: max_depth=5,n_estimators=46, max_features=33, and random_state=1.

At 314, the trained statistical classifier including the firstsub-classifier, the second-sub classifier, and the gatingsub-classifier, is provided for computation of likelihood of malignancyand optionally location of the malignancy within a new single channel 2Dmammographic image. The trained statistical classifier may be locallystored by computing device 204, and/or remotely accessed and/or remotelyobtained from server(s) 218 over network 210.

Reference is now made to FIG. 4 which includes examples of multi channelmammographic images 402-408 computed from single channel mammographicimages, in accordance with some embodiments of the present invention.Multi channel mammographic images 402-408 are shown as false colorenhanced images, based on the red, green, and blue color channels,computed based on the CLAHE method, as described herein. Images 402-404are examples of patches extracted from the multi channel mammographicimages. Images 406-408 are examples of whole multi channel mammographicimages. Images 402 and 406 are examples of images with an indication ofnon-malignancy (i.e., low risk of malignancy). Images 404 and 408 areexamples of images with an indication of malignancy. Image 408 is anexample of an image that includes an indication of benign tissue (i.e.,scarring, which is abnormal breast tissue and may be mistaken for beingmalignancy, when in fact it is benign).

Reference is now made to FIGS. 5A-5B, which includes an example of asingle channel mammographic image and multi channel mammographicimage(s) that are computed from the single channel mammographic image,in accordance with some embodiments of the present invention.Malignancies are more easily visually determined in the multi channelimage in comparison to the single channel image. The strength ofapplication may be controlled by the local malignancy risk score, forexample, simultaneously affording the viewer focus, saliencyvisualization and/or enhanced representation that may improveradiologist performance

FIG. 5A includes a single channel mammographic image 502 and multichannel mammographic images 504-6 that are computed from the singlechannel mammographic image 502, as described herein. Multi channelmammographic images 504-6 are false color enhanced images computed fromblack and white single channel mammographic image 502, by varying theadaptive contract parameters across the color channels, as describedherein. In comparing false color enhanced images 504-6 with black andwhite image 502, it is apparent that the color provides usefulenhancement across a wider range of fidelity resolutions across themajority of the breast.

FIG. 5B includes a single channel mammographic image 552 andcorresponding multi channel mammographic image 554 that is computed fromthe single channel mammographic image 552, as described herein. Multichannel image 556 is another example of a conversion from a singlechannel image.

Reference is now made to FIG. 6, which is a dataflow diagram depictingcomputation of the indication of malignancy for a mammographic image bya trained statistical classifier, in accordance with some embodiments ofthe present invention.

At 602, a 2D single channel mammographic image is converted into themulti channel mammographic image, for example, as described herein.

At 604, optional transformations are performed on the multi channelmammographic image, for example, cropping, resizing, and/or orientation,for example, to conform to inputs of the first neural network (i.e.,sub-classifier), for example, as described herein.

At 606, the multi channel mammographic image is optionally enhanced, forexample, by SPCLAHE.

At 608, a first score is computed by the deep CNN (e.g., Inception v3)processing the multi channel mammographic image, for example, asdescribed herein.

At 610, multiple patches are extracted from the multi channelmammographic image.

At 612, the patches are optionally enhanced, for example, by SPCLAHE.

At 614, optional transformations are performed on the patches, forexample, rotation and/or flipping, for example, to conform to inputs ofthe second neural network (i.e., sub-classifier), for example, asdescribed herein.

At 616, hard negative mining may be performed, for example, as describedherein.

At 618, a second score is computed by the deep CNN (e.g., Inception v3)for each processed patch, for example, as described herein.

It is noted that acts 602-608 may be executed in parallel to acts610-618.

At 620, a random forest classifier computes a final predictionclassification category (e.g., indicative of likelihood of malignancy,indicative of likelihood of non-malignancy, or indicative of likelihoodof a benign abnormality) according to the outputs of the first deep CNN(act 608) and the second deep CNN (act 618), for example, as describedherein.

At 622, exemplary output of the random forest classifier may bepresented, as a probability map and predicted classification category.For example, each patch is associated with a probability of thepredicted classification category.

At 624, a cluster analysis may be performed, by locating pseudoprobabilistic local maxima. For example, patches (which may overlap)associated with indications of likelihood of malignancies are clustered.Local maxima of probabilities for the clustered patches may beidentified. The overlapping region(s) associated with the local maximacluster may be identified as including a likely malignancy.

At 626, a report may be presented, including the classification category(e.g., likelihood of malignancy) and location of the suspectedmalignancy.

Various embodiments and aspects of the present invention as delineatedhereinabove and as claimed in the claims section below find calculatedsupport in the following examples.

EXAMPLES

Reference is now made to the following examples, which together with theabove descriptions illustrate some embodiments of the invention in a nonlimiting fashion.

Inventors performed a computational evaluation according to the systemsand/or methods and/or code instructions described herein, based on thefeatures discussed with reference to FIGS. 1-3, to evaluateclassification of a mammographic image as indicative of malignancy ornot indicative of malignancy. As below in additional detail, the systemsand/or methods and/or code instructions described herein significantlyimproved the sensitivity and/or specificity of automated detection ofmalignancy in mammographic images, in comparison with other automatedmethods, to performance level similar to that of expert radiologists.

Training datasets were created from digital mammography images from TheDigital Database for Screening Mammography (DDSM), which includes morethan 6000 digital mammographic images evenly split between those withand without malignancy as described with reference to M. Heath, K.Bowyer, D. Kopans, R. Moore, and P. Kegelmeyer, “The digital databasefor screening mammography,” Proc. Fifth Int. Work. Digit. Mammogr., pp.212-218, 2001 and the Zebra Mammography Dataset (ZMDS), a dataset of1739 full sized mammograms divided into training, validation and“untouched” test sets comprised of nearly equal numbers of malignant andnon-malignant samples. Both databases represent a random variation ofmammographic breast density. Positive labeled images include malignancy,and negative labeled images represent either normal breast tissue ortissue with definitively benign anomalies. Positive ground truth wasdefined by biopsy proven pathology. Negative samples were defined bybiopsy proven benign tissue and/or at least 2 years of stable imagingfollow up.

First, conversions of the 2D single channel mammographic image bypreprocessing enhancement parameters were evaluated and selected. TheDDSM and ZMDS datasets were used to trained experimental networkdesigns, testing single image sensitivity and specificity against a testset. Initial pathfinding experiments were designed to define the optimalpreprocessing enhancement parameters, testing combinations of parameteralterations as input to ensembles of shallow convolutional neuralnetworks (CNN), for example, as described with reference to A. M.Abdel-Zaher and A. M. Eldeib, “Breast cancer classification using deepbelief networks,” Expert Syst. Appl., vol. 46, pp. 139-144, 2016, and J.Arevalo, F. A. Gonzalez, R. Ramos-Pollán, J. L. Oliveira, and M. A.Guevara Lopez, “Representation learning for mammography mass lesionclassification with convolutional neural networks,” Computer Methods andPrograms in Biomedicine, 2015. The parameter space (the set of allpossible settings for the parameters) of the enhancement parameters,which described the combination of image processes, was searched byevolutionary selection of the ensemble members. Multiple functions wereselected in the vast parameter space, each with a selection of inputsand arguments. The space was permutation dependent with specificfunction ordering, as the output of one function affected eachsubsequent functions.

Most experiments used a maximum of 8 possible preprocessingmanipulations, including empty functions (NoOps), indicating a redundantportion to the genome. The optimal pre-processing combinations wereselected by an evolutionary process: when a new shallow CNN with itsassociated input preprocessing method improved the ensemble'stop1-precision, the network was added as a member. The genome-likedescription of its preprocessing method was added to a pool of parameterstrings from which subsequent candidate members would be bred (theparameter string describing the input preprocessing method, constructedby splicing sections of previously successful preprocessing genomes).

Pathfinding experiments identified Contrast Limited Adaptive HistogramEqualization (CLAHE) as the most useful enhancements during ensembleevolution, yielding a precision of about 86% in classifying malignantfrom non-malignant mammograms. The addition of false color enhancementacross the RGB spectrum was further tested by employing broad windowresolution with low clipping value at the red channel, intermediate atthe green and fine resolution with highest limiting at the blue channel.The false color enhancement resulted in substantially better precisionof about 92% in classifying malignant versus non-malignant mammograms.

Second, the statistical classifier was computed. Parameters wereindependently computed for each of: the neural network that analyzesfull images, the other neural network that analyzes image patches, andthe gating component.

To further increase precision and construct a framework for lesionlocalization, a separate dataset comprising of image patches wasgenerated. Each full breast view yielded approximately 100-400 slidingwindow regions of interest (ROI's), with window strides measuring ⅓ ofthe window's width during experiments. 500×500 pixel ROI's weredownsampled and processed into 299×299 pixel ROI's with RGB colorenhancement (represented as 299×299×3) to conform to the neural networkinput dimensions.

It is noted that unlike the full images (which are more stronglyisometric) the ROIs may be augmented with flips and/or rotations,providing an 8-fold augmentation for the detection window data. Tworounds of hard negative mining were performed to improve specificity ofthe ROI network. It is noted that specificity is particularly importantfor the ROI network as hundreds of inferences occur in each seriesassessment.

Each mammogram underwent pre-processing and enhancement as both a singlefull image and as a set of sliding patches, standardized to 299×299×3,as described herein. Full images and derived patches served as input forthe two respective independent deep CNN model instances describedherein, each based upon the Google™ Inception_v3 model and pre-trainedon ImageNet data.

Inception_v3 was selected to optimize depth, stability and/oravailability of densely pre-trained network snapshots. Atransfer-learning approach was employed, benefiting from featureslearned by lower layers while fine tuning existing snapshots to thepresent mammographic challenge. A deep CNN was computed having broadparameters and with a high resistance to overfitting. Network outputfrom each Inception_v3 instance analysis was inputted into Random Forestclassification code, which processes the combination of the output ofthe full scale based neural network and the patch-based output of theother neural network into a single prediction of a classification class:suspicious for malignancy or non-suspicious for malignancy.

The highest performing gating network was computed based on selection ofthe ROI network outputs (also referred to herein as the secondsub-classifier) and Full Image network outputs (also referred to hereinas the first sub-classifier) to generate the final prediction of theindication of malicious or non-malicious. Multiple classifiers currentlyavailable in SciKit Learn designed to output a pseudo-probabilisticprediction on binary classification were applied, and a course parametersearch was executed on each classifier to identify the best performingclassifier for the gating network. The selected Random Forest classifierincluded the following parameters: max_depth=5, n_estimators=46,max_features=33, random_state=1.

The validation set was used to train the gating network, since thetraining set achieved near 100% accuracy and provides minimal learningopportunity for gating. The untouched test set remained for finalvalidation tests of the statistical classifier that includes the neuralnetwork that analyzes full images, the other neural network thatanalyzes image patches, and the gating component.

Next, the computed statistical classifier was trained from the trainingdatasets, which were split into image with known malignancy and imageswithout malignancy, as described herein. Care was taken not to includeimages from any patient in more than one subset (e.g. for patient withimages in the training set, no images existed in the test or validationsets). The single channel 2D mammographic images were processed tocompute the multi channel false color enhanced images, and the patches,as described herein.

The neural network(s) was initiated with a checkpoint, pre-trained onImagenet, as described with reference to Jia Deng, Wei Dong, R. Socher,Li-Jia Li, Kai Li, and Li Fei-Fei, “ImageNet: A large-scale hierarchicalimage database,” in 2009 IEEE Conference on Computer Vision and PatternRecognition, 2009, pp. 248-255. The full image neural network wastrained on the training set until the value of the loss plateaued.Parameter changes were made by retesting against the validation set(never against the test set), which prevented the risk of informationleakage to the ultimate test set.

The image patches were preprocessed by conversion into the false colorimages and, for the training set only, augmented, as described herein.The ROI network was trained on the augmented, processed image set untilthe loss function plateaued. The ROI network was initiated with acheckpoint, pre-trained on Imagenet. Parameter changes were made byretesting against the validation set.

The random forest gating network was trained, from the outputs of thetwo deep CNNs and the common statistical attributes of the distributionof sliding window scores, which was performed against the validationset.

Final scores were validated for the untouched test set and defined forfull images, ROI (i.e., image patches) and combined data. The scoreswere computed for the untouched test dataset, which included 200positive images and 288 negative images.

Each mammographic image of the untouched test dataset was classified assuspicious or non-suspicious for malignancy based upon the combinationof sliding window ROI scores and the full image score. The overallstand-alone algorithmic accuracy was 84.5%. At a sensitivity of 0.91,specificity was 0.78. The obtained results are similar to thosedescribed for expert radiologists with or without CAD, for example, asdescribed with reference to C. D. Lehman, R. D. Wellman, D. S. M. Buist,K. Kerlikowske, A. N. A. Tosteson, D. L. Miglioretti, and Breast CancerSurveillance Consortium, “Diagnostic Accuracy of Digital ScreeningMammography With and Without Computer-Aided Detection.,” JAMA Intern.Med., vol. 175, no. 11, pp. 1828-37, November 2015. The area under curve(AUC) of the ROC is 0.917, which is similar to values reported forcontemporary single reader Digital Mammography, for example, asdescribed with reference to E. A. Rafferty, J. M. Park, L. E. Philpotts,S. P. Poplack, J. H. Sumkin, E. F. Halpern, and L. T. Niklason,“Assessing radiologist performance using combined digital mammographyand breast tomosynthesis compared with digital mammography alone:results of a multicenter, multireader trial.,” Radiology, vol. 266, no.1, pp. 104-13, January 2013.

Table 1 below summarizes the performance results obtained for thecomputational experiment described herein.

ROI accuracy (per window patch) 0.98 Full image accuracy 0.81 Randomforest accuracy 0.85 Random forest sensitivity 0.91 Random forestspecificity pegged to sensitivity of 0.91 0.78

Table 2 below summarizes the performance results obtained based on thecomputational experiment described herein, for a comparison betweenimages processed according to the systems and/or methods and/or codeinstructions described herein, and images processed by excluding thefeature of conversion from single to multi channel. Table 2 belowprovides evidence that conversion of the image from single channel tomulti channel improves the test accuracy, from 88% to 94%.

Full Image Analysis (Zebra software) Not Enhanced Enhanced (testaccuracy) (test accuracy) 88% 94%

Reference is now made to FIG. 7, which is a receiver operating curve(ROC) depicting the performance results obtained for the computationalexperiment described herein.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

It is expected that during the life of a patent maturing from thisapplication many relevant single channel 2D mammographic images will bedeveloped and the scope of the term single channel 2D mammographicimages intended to include all such new technologies a priori.

As used herein the term “about” refers to ±10%.

The terms “comprises”, “comprising”, “includes”, “including”, “having”and their conjugates mean “including but not limited to”. This termencompasses the terms “consisting of” and “consisting essentially of”.

The phrase “consisting essentially of” means that the composition ormethod may include additional ingredients and/or steps, but only if theadditional ingredients and/or steps do not materially alter the basicand novel characteristics of the claimed composition or method.

As used herein, the singular form “a”, “an” and “the” include pluralreferences unless the context clearly dictates otherwise. For example,the term “a compound” or “at least one compound” may include a pluralityof compounds, including mixtures thereof.

The word “exemplary” is used herein to mean “serving as an example,instance or illustration”. Any embodiment described as “exemplary” isnot necessarily to be construed as preferred or advantageous over otherembodiments and/or to exclude the incorporation of features from otherembodiments.

The word “optionally” is used herein to mean “is provided in someembodiments and not provided in other embodiments”. Any particularembodiment of the invention may include a plurality of “optional”features unless such features conflict.

Throughout this application, various embodiments of this invention maybe presented in a range format. It should be understood that thedescription in range format is merely for convenience and brevity andshould not be construed as an inflexible limitation on the scope of theinvention. Accordingly, the description of a range should be consideredto have specifically disclosed all the possible subranges as well asindividual numerical values within that range. For example, descriptionof a range such as from 1 to 6 should be considered to have specificallydisclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numberswithin that range, for example, 1, 2, 3, 4, 5, and 6. This appliesregardless of the breadth of the range.

Whenever a numerical range is indicated herein, it is meant to includeany cited numeral (fractional or integral) within the indicated range.The phrases “ranging/ranges between” a first indicate number and asecond indicate number and “ranging/ranges from” a first indicate number“to” a second indicate number are used herein interchangeably and aremeant to include the first and second indicated numbers and all thefractional and integral numerals therebetween.

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable subcombination or as suitable in any other describedembodiment of the invention. Certain features described in the contextof various embodiments are not to be considered essential features ofthose embodiments, unless the embodiment is inoperative without thoseelements.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the spirit and broad scopeof the appended claims.

All publications, patents and patent applications mentioned in thisspecification are herein incorporated in their entirety by referenceinto the specification, to the same extent as if each individualpublication, patent or patent application was specifically andindividually indicated to be incorporated herein by reference. Inaddition, citation or identification of any reference in thisapplication shall not be construed as an admission that such referenceis available as prior art to the present invention. To the extent thatsection headings are used, they should not be construed as necessarilylimiting. In addition, any priority document(s) of this applicationis/are hereby incorporated herein by reference in its/their entirety.

What is claimed is:
 1. A method of computing an indication of likelihoodof malignancy in a two dimensional (2D) x-ray based single channelmammographic image by a trained statistical classifier, comprising:receiving a plurality of single channel 2D mammographic images of aplurality of patients; for each one of the plurality of single channel2D mammographic images: receiving a single channel 2D mammographic imageof at least a portion of a breast, wherein the single channel 2Dmammographic image includes a single pixel intensity value for eachpixel of a plurality of pixels thereof; converting the single channel 2Dmammographic image into a multi channel 2D mammographic image includinga plurality of pixel intensity value channels for each pixel of aplurality of pixels thereof; computing by a first sub-classifier of thetrained statistical classifier according to the whole multi channel 2Dmammographic image, a first score indicative of likelihood of malignancywithin the whole multi channel 2D mammographic image; computing by asecond sub-classifier of the trained statistical classifier according toeach respective patch of a plurality of patches extracted from the multichannel 2D mammographic image, a respective second score of a pluralityof second scores indicative of likelihood of malignancy within eachrespective patch of the plurality of patches; computing by a gatingsub-classifier of the trained statistical classifier according to thefirst score and the plurality of second scores, an indication oflikelihood of malignancy and a location of the malignancy; and providingthe indication of likelihood of malignancy and the location of themalignancy, wherein the single channel 2D mammographic image comprises ablack and white image, and the multi channel 2D mammographic imagecomprises a two, three, or four channel false color image; and providingto a remote server for further analysis, a subset of the plurality ofsingle channel 2D mammographic images of the plurality of patientsassociated with likelihood of malignancy.
 2. The method of claim 1,wherein the plurality of single channel 2D mammographic images of theplurality of patients are received from a picture archiving andcommunication system (PACS).
 3. The method of claim 1, wherein theindication of likelihood of malignancy is stored an electronic healthrecord of the respective patient.
 4. The method of claim 3, wherein theindication of likelihood of malignancy is stored in the electronichealth record as a member selected from the group consisting of:metadata, in a predefined field, and in association with themammographic image.
 5. A method of computing an indication of likelihoodof malignancy in a two dimensional (2D) x-ray based single channelmammographic image by a trained statistical classifier, comprising:receiving a plurality of single channel 2D mammographic images of aplurality of patients; for each one of the plurality of single channel2D mammographic images: receiving a single channel 2D mammographic imageof at least a portion of a breast, wherein the single channel 2Dmammographic image includes a single pixel intensity value for eachpixel of a plurality of pixels thereof; converting the single channel 2Dmammographic image into a multi channel 2D mammographic image includinga plurality of pixel intensity value channels for each pixel of aplurality of pixels thereof; computing by a first sub-classifier of thetrained statistical classifier according to the whole multi channel 2Dmammographic image, a first score indicative of likelihood of malignancywithin the whole multi channel 2D mammographic image; computing by asecond sub-classifier of the trained statistical classifier according toeach respective patch of a plurality of patches extracted from the multichannel 2D mammographic image, a respective second score of a pluralityof second scores indicative of likelihood of malignancy within eachrespective patch of the plurality of patches; computing by a gatingsub-classifier of the trained statistical classifier according to thefirst score and the plurality of second scores, an indication oflikelihood of malignancy and a location of the malignancy; and providingthe indication of likelihood of malignancy and the location of themalignancy, wherein the single channel 2D mammographic image comprises ablack and white image, and the multi channel 2D mammographic imagecomprises a two, three, or four channel false color image, whereinsingle channel 2D mammographic images associated with an indication ofrelatively low likelihood of malignancy are further categorized asbenign, indicative of breast tissue with at least one benignabnormality, or categorized as normal, indicative of normal breasttissue.
 6. The method of claim 5, wherein single channel 2D mammographicimages associated with the indication of relatively low likelihood ofmalignancy are not provided for presentation on a display, and singlechannel 2D mammographic images associated with the indication oflikelihood of malignancy are provided for presentation on the display.7. The method of claim 5, wherein single channel 2D mammographic imagesassociated with the indication of relatively low likelihood ofmalignancy are not forwarded to a remote server for further analysis,and single channel 2D mammographic images associated with the indicationof likelihood of malignancy are forwarded to the remote server forfurther analysis.
 8. The method of claim 5, wherein the respectiveindication of each of the single channel 2D mammographic imagesassociated with the indication of relatively low likelihood ofmalignancy is stored in the electronic health record of the respectivepatient.
 9. A method of computing an indication of likelihood ofmalignancy in a two dimensional (2D) x-ray based single channelmammographic image by a trained statistical classifier, comprising:receiving a single channel 2D mammographic image of at least a portionof a breast, wherein the single channel 2D mammographic image includes asingle pixel intensity value for each pixel of a plurality of pixelsthereof; converting the single channel 2D mammographic image into amulti channel 2D mammographic image including a plurality of pixelintensity value channels for each pixel of a plurality of pixelsthereof; computing by a first sub-classifier of the trained statisticalclassifier according to the whole multi channel 2D mammographic image, afirst score indicative of likelihood of malignancy within the wholemulti channel 2D mammographic image; computing by a secondsub-classifier of the trained statistical classifier according to eachrespective patch of a plurality of patches extracted from the multichannel 2D mammographic image, a respective second score of a pluralityof second scores indicative of likelihood of malignancy within eachrespective patch of the plurality of patches; computing by a gatingsub-classifier of the trained statistical classifier according to thefirst score and the plurality of second scores, an indication oflikelihood of malignancy and a location of the malignancy; and providingthe indication of likelihood of malignancy and the location of themalignancy, wherein the indication of likelihood of malignancy and thelocation of the malignancy is provided as a second reader to aninterpreting radiologist, wherein the single channel 2D mammographicimage comprises a black and white image, and the multi channel 2Dmammographic image comprises a two, three, or four channel false colorimage.
 10. The method of claim 9, wherein: (i) the single channel 2Dmammographic image is provided for interpretation by the interpretingradiologist that manually reads the single channel 2D mammographic imagefor visual detection of malignancy, (ii) provided as the second readercomprises providing the indication of likelihood of malignancy and thelocation of the malignancy computed by the trained statisticalclassifier for presentation on a display for reevaluation by theinterpreting radiologist, wherein (i) is performed before (ii).
 11. Themethod of claim 10, further comprising receiving a single interpretationoutcome denoting likelihood of malignancy for the single channel 2Dmammographic image from the interpreting radiologist after thereevaluation by the interpreting radiologist confirms the indication oflikelihood of malignancy and location of malignancy computed by thetrained statistical classifier.
 12. The method of claim 10, furthercomprising computing a single interpretation outcome denoting nomalignancy for the single channel 2D mammographic image when both theindication of likelihood of malignancy computed by the trainedstatistical classifier and the manual reading by the interpretingradiologist denote no malignancy.
 13. A method of computing anindication of likelihood of malignancy in a two dimensional (2D) x-raybased single channel mammographic image by a trained statisticalclassifier, comprising: receiving a single channel 2D mammographic imageof at least a portion of a breast, wherein the single channel 2Dmammographic image includes a single pixel intensity value for eachpixel of a plurality of pixels thereof; converting the single channel 2Dmammographic image into a multi channel 2D mammographic image includinga plurality of pixel intensity value channels for each pixel of aplurality of pixels thereof; computing by a first sub-classifier of thetrained statistical classifier according to the whole multi channel 2Dmammographic image, a first score indicative of likelihood of malignancywithin the whole multi channel 2D mammographic image; computing by asecond sub-classifier of the trained statistical classifier according toeach respective patch of a plurality of patches extracted from the multichannel 2D mammographic image, a respective second score of a pluralityof second scores indicative of likelihood of malignancy within eachrespective patch of the plurality of patches; computing by a gatingsub-classifier of the trained statistical classifier according to thefirst score and the plurality of second scores, an indication oflikelihood of malignancy and a location of the malignancy; providing theindication of likelihood of malignancy and the location of themalignancy; and presenting on a display, at least on of: the computedmulti channel mammographic image, and a visual marking indicative of thelocation of malignancy, wherein the single channel 2D mammographic imagecomprises a black and white image, and the multi channel 2D mammographicimage comprises a two, three, or four channel false color image.
 14. Themethod of claim 13, wherein the visual marking indicative of thelocation of malignancy is presented in association with the presentationon the display of the single channel 2D image.
 15. The method of claim13, wherein the visual marking indicative of the location of malignancyis presented in association with the presentation on the display of thecomputed multi channel mammographic image.
 16. The method of claim 13,wherein the computed likelihood of the detected malignancy may bepresented on the display in association with the multi channelmammographic image.
 17. The method of claim 16, wherein the computedlikelihood comprises a probability of the detected malignancy presentedas a numerical value.
 18. The method of claim 13, wherein the singlechannel 2D mammographic image is presented on the display in associationwith the multi channel mammographic image.
 19. The method of claim 18,wherein the single channel 2D mammographic image is presented on thedisplay located adjacent to the multi channel mammographic image. 20.The method of claim 13, wherein the presented multi channel imageimproves ability of a radiologist to visually detect malignancy in themulti channel image in comparison to the single channel image.